Impaired stationarity perception is associated with increased virtual reality sickness.

Stationarity perception refers to the ability to accurately perceive the surrounding visual environment as world-fixed during self-motion. Perception of stationarity depends on mechanisms that evaluate the congruence between retinal/oculomotor signals and head movement signals. In a series of psychophysical experiments, we systematically varied the congruence between retinal/oculomotor and head movement signals to find the range of visual gains that is compatible with perception of a stationary environment. On each trial, human subjects wearing a head-mounted display execute a yaw head movement and report whether the visual gain was perceived to be too slow or fast. A psychometric fit to the data across trials reveals the visual gain most compatible with stationarity (a measure of accuracy) and the sensitivity to visual gain manipulation (a measure of precision). Across experiments, we varied 1) the spatial frequency of the visual stimulus, 2) the retinal location of the visual stimulus (central vs. peripheral), and 3) fixation behavior (scene-fixed vs. head-fixed). Stationarity perception is most precise and accurate during scene-fixed fixation. Effects of spatial frequency and retinal stimulus location become evident during head-fixed fixation, when retinal image motion is increased. Virtual Reality sickness assessed using the Simulator Sickness Questionnaire covaries with perceptual performance. Decreased accuracy is associated with an increase in the nausea subscore, while decreased precision is associated with an increase in the oculomotor and disorientation subscores.

Motor Signals Mediate Stationarity Perception.

Head movement relative to the stationary environment gives rise to congruent vestibular and visual optic-flow signals. The resulting perception of a stationary visual environment, referred to herein as stationarity perception, depends on mechanisms that compare visual and vestibular signals to evaluate their congruence. Here we investigate the functioning of these mechanisms and their dependence on fixation behavior as well as on the active versus passive nature of the head movement. Stationarity perception was measured by modifying the gain on visual motion relative to head movement on individual trials and asking subjects to report whether the gain was too low or too high. Fitting a psychometric function to the data yields two key parameters of performance. The mean is a measure of accuracy, and the standard deviation is a measure of precision. Experiments were conducted using a head-mounted display with fixation behavior monitored by an embedded eye tracker. During active conditions, subjects rotated their heads in yaw ∼15 deg/s over ∼1 s. Each subject's movements were recorded and played back via rotating chair during the passive condition. During head-fixed and scene-fixed fixation the fixation target moved with the head or scene, respectively. Both precision and accuracy were better during active than passive head movement, likely due to increased precision on the head movement estimate arising from motor prediction and neck proprioception. Performance was also better during scene-fixed than head-fixed fixation, perhaps due to decreased velocity of retinal image motion and increased precision on the retinal image motion estimate. These results reveal how the nature of head and eye movements mediate encoding, processing, and comparison of relevant sensory and motor signals.

Natural statistics of human head orientation constrain models of vestibular processing.

Head orientation relative to gravity determines how gravity-dependent environmental structure is sampled by the visual system, as well as how gravity itself is sampled by the vestibular system. Therefore, both visual and vestibular sensory processing should be shaped by the statistics of head orientation relative to gravity. Here we report the statistics of human head orientation during unconstrained natural activities in humans for the first time, and we explore implications for models of vestibular processing. We find that the distribution of head pitch is more variable than head roll and that the head pitch distribution is asymmetrical with an over-representation of downward head pitch, consistent with ground-looking behavior. We further suggest that pitch and roll distributions can be used as empirical priors in a Bayesian framework to explain previously measured biases in perception of both roll and pitch. Gravitational and inertial acceleration stimulate the otoliths in an equivalent manner, so we also analyze the dynamics of human head orientation to better understand how knowledge of these dynamics can constrain solutions to the problem of gravitoinertial ambiguity. Gravitational acceleration dominates at low frequencies and inertial acceleration dominates at higher frequencies. The change in relative power of gravitational and inertial components as a function of frequency places empirical constraints on dynamic models of vestibular processing, including both frequency segregation and probabilistic internal model accounts. We conclude with a discussion of methodological considerations and scientific and applied domains that will benefit from continued measurement and analysis of natural head movements moving forward.

Natural statistics of human head orientation constrain models of vestibular processing.

Head orientation relative to gravity determines how gravity-dependent environmental structure is sampled by the visual system, as well as how gravity itself is sampled by the vestibular system. Therefore, both visual and vestibular sensory processing should be shaped by the statistics of head orientation relative to gravity. Here we report the statistics of human head orientation during unconstrained natural activities in humans for the first time, and we explore implications for models of vestibular processing. We find that the distribution of head pitch is more variable than head roll and that the head pitch distribution is asymmetrical with an over-representation of downward head pitch, consistent with ground-looking behavior. We further show that pitch and roll distributions can be used as empirical priors in a Bayesian framework to explain previously measured biases in perception of both roll and pitch. We also analyze the dynamics of human head orientation to better understand how gravitational and inertial acceleration are processed by the vestibular system. Gravitational acceleration dominates at low frequencies and inertial acceleration dominates at higher frequencies. The change in relative power of gravitational and inertial components as a function of frequency places empirical constraints on dynamic models of vestibular processing. We conclude with a discussion of methodological considerations and scientific and applied domains that will benefit from continued measurement and analysis of natural head movements moving forward.

Evaluation of the Intel RealSense T265 for tracking natural human head motion.

Accurate and robust tracking of natural human head motion in natural environments is important for a number of applications including virtual and augmented reality, clinical diagnostics, as well as basic scientific research. IMU provide a versatile solution for recording inertial data including linear acceleration and angular velocity, but reconstructing head position is difficult or impossible. This problem can be solved by incorporating visual data using a technique known as visual-inertial simultaneous localization and mapping (VI-SLAM). A recently released commercial solution, the Intel RealSense T265, uses a proprietary VI-SLAM algorithm to estimate linear and angular position and velocity, but the performance of this device for tracking of natural human head motion in natural environments has not yet been comprehensively evaluated against gold-standard methods. In this study, we used a wide range of metrics to evaluate the performance of the T265 with different walking speeds in different environments, both indoor and outdoor, against two gold-standard methods, an optical tracking system and a so-called perambulator. Overall, we find that performance of the T265 relative to these gold-standard methods is most accurate for slow to normal walking speeds in small- to medium-sized environments. The suitability of this device for future scientific studies depends on the application; data presented here can be useful in making that determination.

Positional head-eye tracking outside the lab: an open-source solution.

Simultaneous head and eye tracking has traditionally been confined to a laboratory setting and real-world motion tracking limited to measuring linear acceleration and angular velocity. Recently available mobile devices such as the Pupil Core eye tracker and the Intel RealSense T265 motion tracker promise to deliver accurate measurements outside the lab. Here, the researchers propose a hard- and software framework that combines both devices into a robust, usable, low-cost head and eye tracking system. The developed software is open source and the required hardware modifications can be 3D printed. The researchers demonstrate the system's ability to measure head and eye movements in two tasks: an eyes-fixed head rotation task eliciting the vestibulo-ocular reflex inside the laboratory, and a natural locomotion task where a subject walks around a building outside of the laboratory. The resultant head and eye movements are discussed, as well as future implementations of this system.

Ecological Momentary Assessment of Head Motion: Toward Normative Data of Head Stabilization.

Head stabilization is fundamental for balance during locomotion but can be impaired in elderly or diseased populations. Previous studies have identified several parameters of head stability with possible diagnostic value in a laboratory setting. Recently, the ecological validity of measures obtained in such controlled contexts has been called into question. The aim of this study was to investigate the ecological validity of previously described parameters of head stabilization in a real-world setting. Ten healthy subjects participated in the study. Head and trunk movements of each subject were recorded with inertial measurement units (IMUs) for a period of at least 10 h. Periods of locomotion were extracted from the measurements and predominant frequencies, root mean squares (RMSs) and bout lengths were estimated. As parameters of head stabilization, attenuation coefficients (ACs), harmonic ratios (HRs), coherences, and phase differences were computed. Predominant frequencies were distributed tightly around 2 Hz and ACs, HRs, and coherences exhibited the highest values in this frequency range. All head stability parameters exhibited characteristics consistent with previous reports, although higher variances were observed. These results suggest that head stabilization is tuned to the 2 Hz fundamental frequency of locomotion and that previously described measures of head stability could generalize to a real-world setting. This is the first study to address the ecological validity of these measures, highlighting the potential use of head stability parameters as diagnostic tools or outcome measures for clinical trials. The low cost and ease of use of the IMU technology used in this study could additionally be of benefit for a clinical application.

Towards dynamic modeling of visual-vestibular conflict detection.

Visual-vestibular mismatch is a common occurrence, with causes ranging from vehicular travel, to vestibular dysfunction, to virtual reality displays. Behavioral and physiological consequences of this mismatch include adaptation of reflexive eye movements, oscillopsia, vertigo, and nausea. Despite this significance, we still do not have a good understanding of how the nervous system evaluates visual-vestibular conflict. Here we review research that quantifies perceptual sensitivity to visual-vestibular conflict and factors that mediate this sensitivity, such as noise on visual and vestibular sensory estimates. We emphasize that dynamic modeling methods are necessary to investigate how the nervous system monitors conflict between time-varying visual and vestibular signals, and we present a simple example of a drift-diffusion model for visual-vestibular conflict detection. The model makes predictions for detection of conflict arising from changes in both visual gain and latency. We conclude with discussion of topics for future research.

Vestibulo-Ocular Responses and Dynamic Visual Acuity During Horizontal Rotation and Translation.

Dynamic visual acuity (DVA) provides an overall functional measure of visual stabilization performance that depends on the vestibulo-ocular reflex (VOR), but also on other processes, including catch-up saccades and likely visual motion processing. Capturing the efficiency of gaze stabilization against head movement as a whole, it is potentially valuable in the clinical context where assessment of overall patient performance provides an important indication of factors impacting patient participation and quality of life. DVA during head rotation (rDVA) has been assessed previously, but to our knowledge, DVA during horizontal translation (tDVA) has not been measured. tDVA can provide a valuable measure of how otolith, rather than canal, function impacts visual acuity. In addition, comparison of DVA during rotation and translation can shed light on whether common factors are limiting DVA performance in both cases. We therefore measured and compared DVA during both passive head rotations (head impulse test) and translations in the same set of healthy subjects (n = 7). In addition to DVA, we computed average VOR gain and retinal slip within and across subjects. We observed that during translation, VOR gain was reduced (VOR during rotation, mean ± SD: position gain = 1.05 ± 0.04, velocity gain = 0.97 ± 0.07; VOR during translation, mean ± SD: position gain = 0.21 ± 0.08, velocity gain = 0.51 ± 0.16), retinal slip was increased, and tDVA was worse than during rotation (average rDVA = 0.32 ± 0.15 logMAR; average tDVA = 0.56 ± 0.09 logMAR, p = 0.02). This suggests that reduced VOR gain leads to worse tDVA, as expected. We conclude with speculation about non-oculomotor factors that could vary across individuals and affect performance similarly during both rotation and translation.

Insufficient compensation for self-motion during perception of object speed: The vestibular Aubert-Fleischl phenomenon.

To estimate object speed with respect to the self, retinal signals must be summed with extraretinal signals that encode the speed of eye and head movement. Prior work has shown that differences in perceptual estimates of object speed based on retinal and oculomotor signals lead to biased percepts such as the Aubert-Fleischl phenomenon (AF), in which moving targets appear slower when pursued. During whole-body movement, additional extraretinal signals, such as those from the vestibular system, may be used to transform object speed estimates from a head-centered to a world-centered reference frame. Here we demonstrate that whole-body pursuit in the form of passive yaw rotation, which stimulates the semicircular canals of the vestibular system, leads to a slowing of perceived object speed similar to the classic oculomotor AF. We find that the magnitude of the vestibular and oculomotor AF is comparable across a range of speeds, despite the different types of input signal involved. This covariation might hint at a common modality-independent mechanism underlying the AF in both cases.

Gravity Perception: The Role of the Cerebellum.

The cerebellum is known to support motor behaviors, including postural stability, but new research supports the view that cerebellar function is also critical for perception of spatial orientation, particularly because of its role in vestibular processing.

Psychophysical evidence for auditory motion parallax.

Distance is important: From an ecological perspective, knowledge about the distance to either prey or predator is vital. However, the distance of an unknown sound source is particularly difficult to assess, especially in anechoic environments. In vision, changes in perspective resulting from observer motion produce a reliable, consistent, and unambiguous impression of depth known as motion parallax. Here we demonstrate with formal psychophysics that humans can exploit auditory motion parallax, i.e., the change in the dynamic binaural cues elicited by self-motion, to assess the relative depths of two sound sources. Our data show that sensitivity to relative depth is best when subjects move actively; performance deteriorates when subjects are moved by a motion platform or when the sound sources themselves move. This is true even though the dynamic binaural cues elicited by these three types of motion are identical. Our data demonstrate a perceptual strategy to segregate intermittent sound sources in depth and highlight the tight interaction between self-motion and binaural processing that allows assessment of the spatial layout of complex acoustic scenes.

Optic flow detection is not influenced by visual-vestibular congruency.

Optic flow patterns generated by self-motion relative to the stationary environment result in congruent visual-vestibular self-motion signals. Incongruent signals can arise due to object motion, vestibular dysfunction, or artificial stimulation, which are less common. Hence, we are predominantly exposed to congruent rather than incongruent visual-vestibular stimulation. If the brain takes advantage of this probabilistic association, we expect observers to be more sensitive to visual optic flow that is congruent with ongoing vestibular stimulation. We tested this expectation by measuring the motion coherence threshold, which is the percentage of signal versus noise dots, necessary to detect an optic flow pattern. Observers seated on a hexapod motion platform in front of a screen experienced two sequential intervals. One interval contained optic flow with a given motion coherence and the other contained noise dots only. Observers had to indicate which interval contained the optic flow pattern. The motion coherence threshold was measured for detection of laminar and radial optic flow during leftward/rightward and fore/aft linear self-motion, respectively. We observed no dependence of coherence thresholds on vestibular congruency for either radial or laminar optic flow. Prior studies using similar methods reported both decreases and increases in coherence thresholds in response to congruent vestibular stimulation; our results do not confirm either of these prior reports. While methodological differences may explain the diversity of results, another possibility is that motion coherence thresholds are mediated by neural populations that are either not modulated by vestibular stimulation or that are modulated in a manner that does not depend on congruency.

Visual-Vestibular Conflict Detection Depends on Fixation.

Visual and vestibular signals are the primary sources of sensory information for self-motion. Conflict among these signals can be seriously debilitating, resulting in vertigo [1], inappropriate postural responses [2], and motion, simulator, or cyber sickness [3-8]. Despite this significance, the mechanisms mediating conflict detection are poorly understood. Here we model conflict detection simply as crossmodal discrimination with benchmark performance limited by variabilities of the signals being compared. In a series of psychophysical experiments conducted in a virtual reality motion simulator, we measure these variabilities and assess conflict detection relative to this benchmark. We also examine the impact of eye movements on visual-vestibular conflict detection. In one condition, observers fixate a point that is stationary in the simulated visual environment by rotating the eyes opposite head rotation, thereby nulling retinal image motion. In another condition, eye movement is artificially minimized via fixation of a head-fixed fixation point, thereby maximizing retinal image motion. Visual-vestibular integration performance is also measured, similar to previous studies [9-12]. We observe that there is a tradeoff between integration and conflict detection that is mediated by eye movements. Minimizing eye movements by fixating a head-fixed target leads to optimal integration but highly impaired conflict detection. Minimizing retinal motion by fixating a scene-fixed target improves conflict detection at the cost of impaired integration performance. The common tendency to fixate scene-fixed targets during self-motion [13] may indicate that conflict detection is typically a higher priority than the increase in precision of self-motion estimation that is obtained through integration.

Quantification of Head Movement Predictability and Implications for Suppression of Vestibular Input during Locomotion.

Achieved motor movement can be estimated using both sensory and motor signals. The value of motor signals for estimating movement should depend critically on the stereotypy or predictability of the resulting actions. As predictability increases, motor signals become more reliable indicators of achieved movement, so weight attributed to sensory signals should decrease accordingly. Here we describe a method to quantify this predictability for head movement during human locomotion by measuring head motion with an inertial measurement unit (IMU), and calculating the variance explained by the mean movement over one stride, i.e., a metric similar to the coefficient of determination. Predictability exhibits differences across activities, being most predictable during running, and changes over the course of a stride, being least predictable around the time of heel-strike and toe-off. In addition to quantifying predictability, we relate this metric to sensory-motor weighting via a statistically optimal model based on two key assumptions: (1) average head movement provides a conservative estimate of the efference copy prediction, and (2) noise on sensory signals scales with signal magnitude. The model suggests that differences in predictability should lead to changes in the weight attributed to vestibular sensory signals for estimating head movement. In agreement with the model, prior research reports that vestibular perturbations have greatest impact at the time points and during activities where high vestibular weight is predicted. Thus, we propose a unified explanation for time-and activity-dependent modulation of vestibular effects that was lacking previously. Furthermore, the proposed predictability metric constitutes a convenient general method for quantifying any kind of kinematic variability. The probabilistic model is also general; it applies to any situation in which achieved movement is estimated from both motor signals and zero-mean sensory signals with signal-dependent noise.

Eye Movements in Darkness Modulate Self-Motion Perception.

During self-motion, humans typically move the eyes to maintain fixation on the stationary environment around them. These eye movements could in principle be used to estimate self-motion, but their impact on perception is unknown. We had participants judge self-motion during different eye-movement conditions in the absence of full-field optic flow. In a two-alternative forced choice task, participants indicated whether the second of two successive passive lateral whole-body translations was longer or shorter than the first. This task was used in two experiments. In the first (n = 8), eye movements were constrained differently in the two translation intervals by presenting either a world-fixed or body-fixed fixation point or no fixation point at all (allowing free gaze). Results show that perceived translations were shorter with a body-fixed than a world-fixed fixation point. A linear model indicated that eye-movement signals received a weight of ∼25% for the self-motion percept. This model was independently validated in the trials without a fixation point (free gaze). In the second experiment (n = 10), gaze was free during both translation intervals. Results show that the translation with the larger eye-movement excursion was judged more often to be larger than chance, based on an oculomotor choice probability analysis. We conclude that eye-movement signals influence self-motion perception, even in the absence of visual stimulation.

Dependence of auditory spatial updating on vestibular, proprioceptive, and efference copy signals.

Humans localize sounds by comparing inputs across the two ears, resulting in a head-centered representation of sound-source position. When the head moves, information about head movement must be combined with the head-centered estimate to correctly update the world-centered sound-source position. Spatial updating has been extensively studied in the visual system, but less is known about how head movement signals interact with binaural information during auditory spatial updating. In the current experiments, listeners compared the world-centered azimuthal position of two sound sources presented before and after a head rotation that depended on condition. In the active condition, subjects rotated their head by ∼35° to the left or right, following a pretrained trajectory. In the passive condition, subjects were rotated along the same trajectory in a rotating chair. In the cancellation condition, subjects rotated their head as in the active condition, but the chair was counter-rotated on the basis of head-tracking data such that the head effectively remained fixed in space while the body rotated beneath it. Subjects updated most accurately in the passive condition but erred in the active and cancellation conditions. Performance is interpreted as reflecting the accuracy of perceived head rotation across conditions, which is modeled as a linear combination of proprioceptive/efference copy signals and vestibular signals. Resulting weights suggest that auditory updating is dominated by vestibular signals but with significant contributions from proprioception/efference copy. Overall, results shed light on the interplay of sensory and motor signals that determine the accuracy of auditory spatial updating.

The effect of supine body position on human heading perception.

The use of virtual environments in functional imaging experiments is a promising method to investigate and understand the neural basis of human navigation and self-motion perception. However, the supine position in the fMRI scanner is unnatural for everyday motion. In particular, the head-horizontal self-motion plane is parallel rather than perpendicular to gravity. Earlier studies have shown that perception of heading from visual self-motion stimuli, such as optic flow, can be modified due to visuo-vestibular interactions. With this study, we aimed to identify the effects of the supine body position on visual heading estimation, which is a basic component of human navigation. Visual and vestibular heading judgments were measured separately in 11 healthy subjects in upright and supine body positions. We measured two planes of self-motion, the transverse and the coronal plane, and found that, although vestibular heading perception was strongly modified in a supine position, visual performance, in particular for the preferred head-horizontal (i.e., transverse) plane, did not change. This provides behavioral evidence in humans that direction estimation from self-motion consistent optic flow is not modified by supine body orientation, demonstrating that visual heading estimation is one component of human navigation that is not influenced by the supine body position required for functional brain imaging experiments.

Comparison of linear motion perception thresholds in vestibular migraine and Menière's disease.

Linear motion perceptual thresholds (PTs) were compared between patients with Menière's disease (MD) and vestibular migraine (VM). Twenty patients with VM, 27 patients with MD and 34 healthy controls (HC) were examined. PTs for linear motion along the inter-aural (IA), naso-occipital axes (NO), and head-vertical (HV) axis were measured using a multi-axis motion platform. Ocular and cervical vestibular evoked myogenic potentials (o/c VEMP) were performed and the dizziness handicap inventory (DHI) administered. In order to discriminate between VM and MD, we also evaluated the diagnostic accuracy of applied methods. PTs depended significantly on the group tested (VM, MD and HC), as revealed by ANCOVA with group as the factor and age as the covariate. This was true for all motion axes (IA, HV and NO). Thresholds were highest for MD patients, significantly higher than for all other groups for all motion axes, except for the IA axis when compared with HC group suggesting decreased otolith sensitivity in MD patients. VM patients had thresholds that were not different from those of HC, but were significantly lower than those of the MD group for all motion axes. The cVEMP p13 latencies differed significantly across groups being lowest in VM. There was a statistically significant association between HV and NO thresholds and cVEMP PP amplitudes. Diagnostic accuracy was highest for the IA axis, followed by cVEMP PP amplitudes, NO and HV axes. To conclude, patients with MD had significantly higher linear motion perception thresholds compared to patients with VM and controls. Except for reduced cVEMP latency, there were no differences in c/oVEMP between MD, VM and controls.

Multisensory self-motion compensation during object trajectory judgments.

Judging object trajectory during self-motion is a fundamental ability for mobile organisms interacting with their environment. This fundamental ability requires the nervous system to compensate for the visual consequences of self-motion in order to make accurate judgments, but the mechanisms of this compensation are poorly understood. We comprehensively examined both the accuracy and precision of observers' ability to judge object trajectory in the world when self-motion was defined by vestibular, visual, or combined visual-vestibular cues. Without decision feedback, subjects demonstrated no compensation for self-motion that was defined solely by vestibular cues, partial compensation (47%) for visually defined self-motion, and significantly greater compensation (58%) during combined visual-vestibular self-motion. With decision feedback, subjects learned to accurately judge object trajectory in the world, and this generalized to novel self-motion speeds. Across conditions, greater compensation for self-motion was associated with decreased precision of object trajectory judgments, indicating that self-motion compensation comes at the cost of reduced discriminability. Our findings suggest that the brain can flexibly represent object trajectory relative to either the observer or the world, but a world-centered representation comes at the cost of decreased precision due to the inclusion of noisy self-motion signals.